Papers with text-video retrieval
Normalized Contrastive Learning for Text-Video Retrieval (2022.emnlp-main)
Copied to clipboard
| Challenge: | Cross-modal contrastive learning suffers from incorrect normalization of the sum retrieval probabilities of each text or video instance. |
| Approach: | They propose a normalized contrastive learning algorithm that normalizes the sum retrieval probabilities of each instance so that every text and video instance is fairly represented. |
| Outcome: | Empirical results show that NCL brings significant gains in text-video retrieval on different model architectures without any architecture engineering. |
ORANGE: Text-video Retrieval via Watch-time-aware Heterogeneous Graph Contrastive Learning (2023.emnlp-industry)
Copied to clipboard
Yucheng Lin, Tim Chang, Yaning Chang, Jianqiang Ma, Donghui Li, Ting Peng, Zang Li, Zhiyi Zhou, Feng Wang
| Challenge: | Existing methods for text-video retrieval focus on informative representations and delicate matching mechanisms, but real-world scenarios often involve brief, ambiguous queries and low-quality videos. |
| Approach: | They propose a novel method to learn informative embeddings for queries and videos . they use a watch-time-aware contrastive learning paradigm to capture dependencies . |
| Outcome: | The proposed method is effective in a real-world video-search service. |
MERLIN: Multimodal Embedding Refinement via LLM-based Iterative Navigation for Text-Video Retrieval-Rerank Pipeline (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent advances in text-video retrieval neglect the crucial user perspective, leading to discrepancies between user queries and content retrieved. |
| Approach: | They propose a novel, training-free pipeline that leverages Large Language Models for iterative feedback learning. |
| Outcome: | Experimental results show that MERLIN significantly outperforms existing systems in video retrieval. |
Contrastive Video-Language Learning with Fine-grained Frame Sampling (2022.aacl-main)
Copied to clipboard
| Challenge: | despite recent progress in video and language representation learning, the weak or sparse correspondence between the two modalities remains a bottleneck. |
| Approach: | They propose a fine-grained contrastive objective for video frame sampling to improve cross-modal correspondence. |
| Outcome: | The proposed approach achieves state-of-the-art performance on YouCookII with long videos. |
GHAN: Graph-Based Hierarchical Aggregation Network for Text-Video Retrieval (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to text-video retrieval are limited due to structural and semantic differences between text and video. |
| Approach: | They propose an end-to-end graph-based hierarchical aggregation network for text-video retrieval according to the hierarchy possessed by text and video. |
| Outcome: | The proposed model achieves Recall@1 of 73.0%, 65.6%, and 64.0% better than the current state-of-the-art model. |
LiteVL: Efficient Video-Language Learning with Enhanced Spatial-Temporal Modeling (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent large-scale video-language pre-trained models have shown appealing performance on downstream tasks. |
| Approach: | They propose a video-text model that adapts a pre-trained image-language model into a text-based model without heavy pre-training. |
| Outcome: | The proposed model outperforms existing models on video-text retrieval and video question answering tasks without heavy pre-training. |
Captioning for Text-Video Retrieval via Dual-Group Direct Preference Optimization (2025.findings-emnlp)
Copied to clipboard
| Challenge: | auxiliary captions are generic and indistinguishable across visually similar videos . conventional captioning approaches are evaluated using language relevance scores . |
| Approach: | They propose a retrieval framework that directly optimizes caption generation using retrieval relevance scores. |
| Outcome: | The proposed retrieval framework optimizes caption generation using retrieval relevance scores . dual-group direct preference optimization is a learning strategy that supervises captioning . |